Tea, Earl Grey, Hot: Designing Speech Interactions from the Imagined Ideal of Star Trek


Speech is now common in daily interactions with our devices, thanks to voice user interfaces (VUIs) like Alexa. Despite their seeming ubiquity, designs often do not match users’ expectations. Science fiction, which is known to influence design of new technologies, has included VUIs for decades. Star Trek: The Next Generation is a prime example of how people envisioned ideal VUIs. Understanding how current VUIs live up to Star Trek’s utopian technologies reveals mismatches between current designs and user expectations, as informed by popular fiction. Combining conversational analysis and VUI user analysis, we study voice interactions with the Enterprise’s computer and compare them to current interactions. Independent of futuristic computing power, we find key design-based differences: Star Trek interactions are brief and functional, not conversational, they are highly multimodal and context-driven, and there is often no spoken computer response. From this, we suggest paths to better align VUIs with user expectations.

Source: https://github.com/rfordatascience/tidytuesday/blob/master/data/2021/2021-08-17/readme.md

Chart 1: Lauren

Chart 2: Kim

Chart 3: Kayna


When we interact with voice-command technology, we use certain types of interactions to ‘wake’ the system (“Hey Siri…”), ‘command’ the system (“Play a song on Spotify”), ‘question’ the system (“What is the temperature for today?”), and many other types of interactions.

When our technology or system interacts with us, it uses similar types of interactions to ‘clarify’ (“What song would you like to hear?”), indicate that something is ‘in progress’ (“Let me find that for you…”), have ‘conversation’ (“You are welcome”), and other types of interactions.

On the Starship Enterprise, the crew interacts with the computer, and the computer interacts with the crew, through different ‘Interaction Types’. These are organized by person (the person interacting with the computer) and by computer (the computer interacting with the person). Definitions and examples of these interaction types can be found below.

By Person:

Rank Person Definition Examples
1 Command Utterances that directly tell the computer what to do. Run a diagnostic on the port nacelle.
2 Question Utterances that ask the computer for something. Where is Captain Picard?
3 Statement Utterances tell don’t tell the computer or ask it, but meaning is inferred. Deck four. I wish to learn about Earth.
4 Password Utterances that contain a password. This is Captain Picard.
5 Wake Word Key phrases used to activate the computer. Computer. Holodeck.
6 Comment Utterances that have no intended action for the computer. Excellent. Ferrazene has a complex molecular structure.
7 Conversation Utterances that are more like human conversation, such as phatic espressions, formalities, and colloquial speech. Well, check it again! Then run it for us, dear.

By Computer:

Rank Person Definition Examples
1 Clarification Utterances asking for more information What temperature?
2 Response Utterances that respond to a person’s query or action. Here is the information you requested.
3 System Alert Utterances that either respond to a person’s requested action (often a warning) or are prompted by the system, rather than a person. That is not recommended. Outer hull breach.
4 Information Utterances that inform users of what is happening without being prompted to do so (which would be a response). Opening main shuttle bay doors.
5 Countdown Utterances that are part of a countdown Self destruct in 30 seconds. Self destruct in 20 seconds.
6 In Progress Utterances that indicates that a process is ongoing or updates on that process. Accessing.
7 Conversation Utterances that are more like human conversation, such as phatic espressions, formalities, and colloquial speech. You’re more than welcome, Commander Riker.

Because the computer on the Starship Enterprise can generate objects and display information without responding, it is of interest to examine the proportion of occurrences when the computer responds non-verbally or with actions (versus with ‘By Computer’ interactions). The visualization to the left shows frequency of each interaction type by person, and frequency of subsequent interaction by computer via verbal and non-verbal response rates.

Source: http://www.speechinteraction.org/TNG/TeaEarlGreyHotDatasetCodeBook.pdf

Chart 4: Stacey

Chart 5: Courtney

---
title: "Untitled"
output: 
  flexdashboard::flex_dashboard:
    storyboard: true
    social: menu
    source: embed
---

```{r setup, include=FALSE}
library(flexdashboard)
library(readr)
library(knitr)
startrek <- read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-08-17/computer.csv')

```


### Tea, Earl Grey, Hot: Designing Speech Interactions from the Imagined Ideal of Star Trek

```{r}
include_graphics('https://raw.githubusercontent.com/LaurS12/ERHS535_Group_Project/main/Images/data_description.png')

#note: this looks like garbage in the markdown file, but if you knit, it shows up correct.
```

***

Speech is now common in daily interactions with our devices, thanks to voice user interfaces (VUIs) like Alexa. Despite their seeming ubiquity, designs often do not match users’ expectations. Science fiction, which is known to influence design of new technologies, has included VUIs for decades. Star Trek: The Next Generation is a prime example of how people envisioned ideal VUIs. Understanding how current VUIs live up to Star Trek’s utopian technologies reveals mismatches between current designs and user expectations, as informed by popular fiction. Combining conversational analysis and VUI user analysis, we study voice interactions with the Enterprise’s computer and compare them to current interactions. Independent of futuristic computing power, we find key design-based differences: Star Trek interactions are brief and functional, not conversational, they are highly multimodal and context-driven, and there is often no spoken computer response. From this, we suggest paths to better align VUIs with user expectations.

Source: https://github.com/rfordatascience/tidytuesday/blob/master/data/2021/2021-08-17/readme.md


### Chart 1: Lauren

```{r}

```

### Chart 2: Kim

```{r}

```

### Chart 3: Kayna

```{r}

```

***

When we interact with voice-command technology, we use certain types of interactions to 'wake' the system ("Hey Siri..."), 'command' the system ("Play a song on Spotify"), 'question' the system ("What is the temperature for today?"), and many other types of interactions. 

When our technology or system interacts with us, it uses similar types of interactions to 'clarify' ("What song would you like to hear?"), indicate that something is 'in progress' ("Let me find that for you..."), have 'conversation' ("You are welcome"), and other types of interactions.

On the Starship Enterprise, the crew interacts with the computer, and the computer interacts with the crew, through different 'Interaction Types'. These are organized by person (the person interacting with the computer) and by computer (the computer interacting with the person). Definitions and examples of these interaction types can be found below. 

By Person:

| Rank | Person       | Definition                                                                                                        | Examples                                                |
|------|--------------|-------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| 1    | Command      | Utterances that directly tell the computer what to do.                                                            | Run a diagnostic on the port nacelle.                   |
| 2    | Question     | Utterances that ask the computer for something.                                                                   | Where is Captain Picard?                                |
| 3    | Statement    | Utterances tell don't tell the computer or ask it, but meaning is inferred.                                       | Deck four. I wish to learn about Earth.                 |
| 4    | Password     | Utterances that contain a password.                                                                               | This is Captain Picard.                                 |
| 5    | Wake Word    | Key phrases used to activate the computer.                                                                        | Computer. Holodeck.                                     |
| 6    | Comment      | Utterances that have no intended action for the computer.                                                         | Excellent. Ferrazene has a complex molecular structure. |
| 7    | Conversation | Utterances that are more like human conversation, such as phatic espressions, formalities, and colloquial speech. | Well, check it again! Then run it for us, dear.         |


By Computer:

| Rank | Person        | Definition                                                                                                                           | Examples                                                  |
|------|---------------|--------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|
| 1    | Clarification | Utterances asking for more information                                                                                               | What temperature?                                         |
| 2    | Response      | Utterances that respond to a person's query or action.                                                                               | Here is the information you requested.                    |
| 3    | System Alert  | Utterances that either respond to a person's requested action (often a warning) or are prompted by the system, rather than a person. | That is not recommended. Outer hull breach.               |
| 4    | Information   | Utterances that inform users of what is happening without being prompted to do so (which would be a response).                       | Opening main shuttle bay doors.                           |
| 5    | Countdown     | Utterances that are part of a countdown                                                                                              | Self destruct in 30 seconds. Self destruct in 20 seconds. |
| 6    | In Progress   | Utterances that indicates that a process is ongoing or updates on that process.                                                      | Accessing.                                                |
| 7    | Conversation  | Utterances that are more like human conversation, such as phatic espressions, formalities, and colloquial speech.                    | You're more than welcome,  Commander Riker.               |

Because the computer on the Starship Enterprise can generate objects and display information without responding, it is of interest to examine the proportion of occurrences when the computer responds non-verbally or with actions (versus with 'By Computer' interactions). The visualization to the left shows frequency of each interaction type by person, and frequency of subsequent interaction by computer via verbal and non-verbal response rates.

Source: http://www.speechinteraction.org/TNG/TeaEarlGreyHotDatasetCodeBook.pdf

### Chart 4: Stacey

```{r}

```

### Chart 5: Courtney

```{r}
# Packages
library(wordcloud)
library(RColorBrewer)
library(wordcloud2)
library(tm)
library(tidyverse)

# Filter to necessary column
text <- startrek$interaction

# Clean text
docs <- Corpus(VectorSource(text))

docs <- docs %>%
  tm_map(removeNumbers) %>%
  tm_map(removePunctuation) %>%
  tm_map(stripWhitespace)

docs <- tm_map(docs, content_transformer(tolower))

docs <- tm_map(docs, removeWords, stopwords("english"))

# Create matrix with counts
dtm <- TermDocumentMatrix(docs)

matrix <- as.matrix(dtm) 

words <- sort(rowSums(matrix),decreasing = TRUE) 

df <- data.frame(word = names(words), freq = words)

# Wordcloud
wordcloud2(data=df, size=1.6, color='random-dark', shape = "cardoid")

```